This project aims to explore data on housing prices in an effort to design a model for estimating pricing and other behaviors of the housing market, specifically within King County.
load in the required packages
#load dataset from project folder
suppressWarnings({housing_data <- as.tbl(read.csv("kc_house_data.csv", stringsAsFactors = F))})
#remove unused columns and normalize the data
housing_data_rev <- housing_data[, 3:21]
load in dataset and remove unused features
This dataset is very large, so in order to get a better visual representation of the distribution of the data, a sample of 1000 randomly chosen rows is used instead of the whole dataset (ordered by price, as this is the dependent variable). The sample indexes are selected at runtime every time the script runs, so the data and analysis are different each time the script is run. Because the sampling is random and the dataset is homogenous and static, the 1000 rows chosen should always be representative of the nature of the whole dataset.
#Look at the distributions:
p1 <- plot_ly() %>% add_markers(name = "Pricing",
x = 1:length(housing_train_data$price),
y = housing_train_data$price) %>%
layout(title = "King County Housing Feature Distributions")
p2 <- plot_ly() %>% add_markers(name = "Rating v Pricing",
x = housing_train_data$grade,
y = housing_train_data$price)
p3 <- plot_ly() %>% add_markers(name = "Condition v Pricing",
x = housing_train_data$condition,
y = housing_train_data$price)
p4 <- plot_ly() %>% add_markers(name = "Sq. Footage v Pricing",
x = housing_train_data$sqft_lot,
y = housing_train_data$price)
subplot(p1, p2, p3, p4, nrows = 2)
Looking at the distributions of prices over their ordering by magnitude, it appears that the prices of houses that are below $1-2 million follow a linear decline, with all houses above $1-2 million increasing sharply in their prices and bucking the trend. Assuming that the features in this dataset capture most of the factors that influence the pricing of a home, some linear combination of the features could give a reasonable estimate for the pricing of an individual house below $1-2 million. Since the distribution is closer to a power law, two seperate linear functions should give a pretty good estimate by accounting for the ‘knee’ in the pricing distribution. First though, we’ll try fitting a single line to the data and see what kind of error comes out.
#Scale datasets
housing_train_data_scaled <- cbind(housing_train_data[,1], scale(housing_train_data[,2:7]))
housing_test_data_scaled <- cbind(housing_test_data[,1], scale(housing_test_data[,2:7]))
summary(housing_train_data_scaled)
price sqft_living sqft_lot condition grade yr_built yr_renovated
Min. : 78000 Min. :-1.9425 Min. :-0.3584 Min. :-3.6637 Min. :-5.6012 Min. :-2.4072 Min. :-0.2145
1st Qu.: 324950 1st Qu.:-0.7100 1st Qu.:-0.2460 1st Qu.:-0.6249 1st Qu.:-0.5509 1st Qu.:-0.6765 1st Qu.:-0.2145
Median : 452000 Median :-0.1803 Median :-0.1809 Median :-0.6249 Median :-0.5509 Median : 0.1379 Median :-0.2145
Mean : 545343 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000
3rd Qu.: 649612 3rd Qu.: 0.5116 3rd Qu.:-0.1033 3rd Qu.: 0.8945 3rd Qu.: 0.2908 3rd Qu.: 0.8844 3rd Qu.:-0.2145
Max. :7700000 Max. :12.3818 Max. :41.0458 Max. : 2.4138 Max. : 4.4994 Max. : 1.4953 Max. : 4.7073
summary(housing_test_data_scaled)
price sqft_living sqft_lot condition grade yr_built yr_renovated
Min. : 83000 Min. :-1.8818 Min. :-0.4441 Min. :-3.6826 Min. :-3.9686 Min. :-2.5008 Min. :-0.1987
1st Qu.: 320000 1st Qu.:-0.6862 1st Qu.:-0.3053 1st Qu.:-0.6176 1st Qu.:-0.5599 1st Qu.:-0.6673 1st Qu.:-0.1987
Median : 445000 Median :-0.1674 Median :-0.2243 Median :-0.6176 Median :-0.5599 Median : 0.1284 Median :-0.1987
Mean : 533756 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000 Mean : 0.0000
3rd Qu.: 631719 3rd Qu.: 0.5319 3rd Qu.:-0.1155 3rd Qu.: 0.9149 3rd Qu.: 0.2923 3rd Qu.: 0.8895 3rd Qu.:-0.1987
Max. :3278000 Max. : 5.9343 Max. :13.6084 Max. : 2.4474 Max. : 3.7010 Max. : 1.4776 Max. : 5.0703
require(tensorflow)
#split data for training
#test general linear model on relevant quantitative data
#Test on a nueral network of 10 nodes
X <- tf$placeholder(tf$float32, shape(NULL, 7L))
Y <- tf$placeholder(tf$float32, shape(NULL, 1L))
W1 <- tf$Variable(tf$zeros(shape(10, 7)))
b1 <- tf$Variable(tf$zeros(shape(10, 1)))
W2 <- tf$Variable(tf$zeros(shape(1, 10)))
b2 <- tf$Variable(tf$zeros(shape(1, 1)))
A1 <- tf$nn$relu(tf$add(tf$matmul(X,W1), b1))
A2 <- tf$nn$relu(tf$add(tf$matmul(A1,W2), b2))
cost <- tf
tf$placeholder(tf$float32, shape(7L, NULL))
housing_train_price <- housing_train_data[, "price"]
housing_test_price <- housing_test_data[, "price"]
glm_mod <- rxFastLinear(price ~ sqft_living + grade + condition, data = housing_train_data_scaled, type = "regression", verbose = 0)
price_mod <- rxPredict(glm_mod, data = housing_test_data_scaled, verbose = 0, extraVarsToWrite = c("sqft_living", "grade","condition"))
xvals <- 1:length(price_mod$Score)
price_est_plotly <- plot_ly() %>%
add_markers(name = "Test Set Price", x = xvals, y = housing_test_price$price) %>%
add_markers(name = "Linear Model est", x = xvals, y = price_mod$Score) %>%
add_lines(name = "Magnitude Difference", x = xvals, y = price_mod$Score - housing_test_price$price) %>%
layout(title = "Attempting Linear Regression Model
</br> for Pricing Homes in King County")
#rxLinePlot(Score ~ sqft_living + grade + condition, type = c("p", "r"), data = price_mod)
price_est_plotly
#summary(glm_mod)
Given that the distribution of prices was already known to follow a non-linear distribution, the result of this linear estimate isn’t very surprising. The linear fit may have been within a few hundred thousand dollars for the long tail, but the whole seems to be corrupted in trying to fit the collection of values above the elbow. Below around $1 million dollars, the housing prices are pretty evenly distributed, but as the prices increase they start jumping up by orders of magnitude. Lets try the same approach for all values of price < $600K
housing_train_data_rev1 <- filter(housing_train_data_scaled, price < 8e5)
housing_train_price_rev1 <- filter(housing_train_price, price < 8e5)
housing_test_data_rev1 <- filter(housing_test_data_scaled, price < 8e5)
housing_test_price_rev1 <- filter(housing_test_price, price < 8e5)
max_test_rev1 <- apply(housing_test_data_rev1, FUN = max, MARGIN = 2)
min_test_rev1 <- apply(housing_test_data_rev1, FUN = min, MARGIN = 2)
mean_test_rev1 <- apply(housing_test_data_rev1, FUN = mean, MARGIN = 2)
glm_mod_rev1 <- rxFastLinear(price ~ sqft_living + grade + condition, data = housing_train_data_rev1, type = "regression", verbose = 0)
price_mod_rev1 <- rxPredict(glm_mod_rev1, data = housing_test_data_rev1, verbose = 0, extraVarsToWrite = c("sqft_living", "grade","condition"))
xvals_rev1 <- 1:length(price_mod_rev1$Score)
price_est_plotly_rev1 <- plot_ly() %>%
add_markers(name = "Test Set Price", x = xvals_rev1, y = housing_test_price_rev1$price) %>%
add_markers(name = "Linear Model est", x = xvals_rev1, y = price_mod_rev1$Score) %>%
add_lines(name = "Magnitude Difference", x = xvals_rev1, y = price_mod_rev1$Score - housing_test_price_rev1$price) %>%
layout(title = "Attempting Linear Regression Model
</br>for Pricing Homes in King County
</br>Where Price of Home < $800k")
price_est_plotly_rev1
#summary(glm_mod_rev1)
By Cutting off the less linear part of the dataset, the maxerror between the estimate and the actual price was decreased.
LS0tDQp0aXRsZTogIktpbmcgQ291bnR5IEhvdXNpbmcgUHJpY2VzIg0KYXV0aG9yOiAiRHVuY2FuIE1jS2lubm9uIg0Kb3V0cHV0OiBodG1sX25vdGVib29rDQotLS0NCg0KVGhpcyBwcm9qZWN0IGFpbXMgdG8gZXhwbG9yZSBkYXRhIG9uIGhvdXNpbmcgcHJpY2VzIGluIGFuIGVmZm9ydCB0byBkZXNpZ24gYSBtb2RlbCBmb3IgZXN0aW1hdGluZyBwcmljaW5nIGFuZCBvdGhlciBiZWhhdmlvcnMgb2YgdGhlIGhvdXNpbmcgbWFya2V0LCBzcGVjaWZpY2FsbHkgd2l0aGluIEtpbmcgQ291bnR5LiANCg0KDQpgYGB7ciBtZXNzYWdlID0gRiwgd2FybmluZyA9IEYsIGVjaG8gPSBGLCBzdHJpcC53aGl0ZSA9IEYsIHRpZHkgPSBUfQ0KDQpzdXBwcmVzc1BhY2thZ2VTdGFydHVwTWVzc2FnZXMoDQogIHsNCiAgICAjaW5zdGFsbCBvciBsb2FkIHJlcXVpcmVkIHBhY2thZ2VzIGludG8gdGhlIHdvcmtzcGFjZQ0KICAgIHBhY2thZ2VfbGlzdCA8LSBjKCd0aWR5dmVyc2UnLCAna25pdHInLCAncGxvdGx5JywnUkNvbG9yQnJld2VyJywnc3FsZGYnLCAnc3RyaW5ncicsJ3RlbnNvcmZsb3cnKQ0KICAgIG5vbl9pbnN0YWxsZWQgPC0gcGFja2FnZV9saXN0WyEocGFja2FnZV9saXN0ICVpbiUgaW5zdGFsbGVkLnBhY2thZ2VzKClbLCJQYWNrYWdlIl0pXQ0KICAgIGlmKGxlbmd0aChub25faW5zdGFsbGVkKSkgaW5zdGFsbC5wYWNrYWdlcyhub25faW5zdGFsbGVkKQ0KICAgIHJlcXVpcmUoJ2RwbHlyJykNCiAgICByZXF1aXJlKCdwbG90bHknKQ0KICAgIHJlcXVpcmUoJ1JDb2xvckJyZXdlcicpDQogICAgcmVxdWlyZSgnc3FsZGYnKQ0KICAgIHJlcXVpcmUoJ2tuaXRyJykNCiAgICByZXF1aXJlKCdzdHJpbmdyJykNCiAgICByZXF1aXJlKCd0ZW5zb3JmbG93JykNCiAgfQ0KKQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KG1lc3NhZ2UgPSBGLCB3YXJuaW5nID0gRiwgc3RyaXAud2hpdGUgPSBGLCB0aWR5ID0gVCkNCmBgYA0KbG9hZCBpbiB0aGUgcmVxdWlyZWQgcGFja2FnZXMNCg0KDQpgYGB7cn0NCiNsb2FkIGRhdGFzZXQgZnJvbSBwcm9qZWN0IGZvbGRlcg0Kc3VwcHJlc3NXYXJuaW5ncyh7aG91c2luZ19kYXRhIDwtIGFzLnRibChyZWFkLmNzdigia2NfaG91c2VfZGF0YS5jc3YiLCBzdHJpbmdzQXNGYWN0b3JzID0gRikpfSkNCg0KDQojcmVtb3ZlIHVudXNlZCBjb2x1bW5zIGFuZCBub3JtYWxpemUgdGhlIGRhdGENCmhvdXNpbmdfZGF0YV9yZXYgPC0gaG91c2luZ19kYXRhWywgMzoyMV0NCg0KDQojdXNpbmcgZHBseXIgc2FtcGxpbmcgdG8gY29sbGVjdCBhbmQgcmFuZG9tIHNhbXBsaW5ncw0KaG91c2luZ190cmFpbl9kYXRhIDwtIGhvdXNpbmdfZGF0YV9yZXYgJT4lIHNlbGVjdChjKDEsNCw1LDksMTAsMTMsMTQpKSAlPiUgc2FtcGxlX24oMTAwMDApICU+JQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFycmFuZ2UoZGVzYyhwcmljZSkpDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCmhvdXNpbmdfdGVzdF9kYXRhIDwtIGhvdXNpbmdfZGF0YV9yZXYgJT4lIHNlbGVjdChjKDEsNCw1LDksMTAsMTMsMTQpKSAlPiUgc2FtcGxlX24oMjAwMCkgJT4lDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhcnJhbmdlKGRlc2MocHJpY2UpKQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KDQpwcmljZV9zdGF0cyA8LSBsaXN0KCJ0cmFpbl9tZWFuIiA9IG1lYW4oaG91c2luZ190cmFpbl9kYXRhJHByaWNlKSwgDQogICAgICAgICAgICAgICAgICAgICJ0cmFpbl9TRCIgPSBzZChob3VzaW5nX3RyYWluX2RhdGEkcHJpY2UpLA0KICAgICAgICAgICAgICAgICAgICAidGVzdF9tZWFuIiA9IG1lYW4oaG91c2luZ190ZXN0X2RhdGEkcHJpY2UpLA0KICAgICAgICAgICAgICAgICAgICAidGVzdF9TRCIgPSBzZChob3VzaW5nX3Rlc3RfZGF0YSRwcmljZSkpDQoNCnByaWNlX3N0YXRzDQpgYGANCmxvYWQgaW4gZGF0YXNldCBhbmQgcmVtb3ZlIHVudXNlZCBmZWF0dXJlcw0KDQoNClRoaXMgZGF0YXNldCBpcyB2ZXJ5IGxhcmdlLCBzbyBpbiBvcmRlciB0byBnZXQgYSBiZXR0ZXIgdmlzdWFsIHJlcHJlc2VudGF0aW9uIG9mIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIGRhdGEsIGEgc2FtcGxlIG9mIDEwMDAgcmFuZG9tbHkgY2hvc2VuIHJvd3MgaXMgdXNlZCBpbnN0ZWFkIG9mIHRoZSB3aG9sZSBkYXRhc2V0IChvcmRlcmVkIGJ5IHByaWNlLCBhcyB0aGlzIGlzIHRoZSBkZXBlbmRlbnQgdmFyaWFibGUpLiAgVGhlIHNhbXBsZSBpbmRleGVzIGFyZSBzZWxlY3RlZCBhdCBydW50aW1lIGV2ZXJ5IHRpbWUgdGhlIHNjcmlwdCBydW5zLCBzbyB0aGUgZGF0YSBhbmQgYW5hbHlzaXMgYXJlIGRpZmZlcmVudCBlYWNoIHRpbWUgdGhlIHNjcmlwdCBpcyBydW4uIEJlY2F1c2UgdGhlIHNhbXBsaW5nIGlzIHJhbmRvbSBhbmQgdGhlIGRhdGFzZXQgaXMgaG9tb2dlbm91cyBhbmQgc3RhdGljLCB0aGUgMTAwMCByb3dzIGNob3NlbiBzaG91bGQgYWx3YXlzIGJlIHJlcHJlc2VudGF0aXZlIG9mIHRoZSBuYXR1cmUgb2YgdGhlIHdob2xlIGRhdGFzZXQuDQoNCg0KYGBge3J9DQojTG9vayBhdCB0aGUgZGlzdHJpYnV0aW9uczoNCnAxIDwtIHBsb3RfbHkoKSAlPiUgYWRkX21hcmtlcnMobmFtZSA9ICJQcmljaW5nIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeCA9IDE6bGVuZ3RoKGhvdXNpbmdfdHJhaW5fZGF0YSRwcmljZSksIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ID0gaG91c2luZ190cmFpbl9kYXRhJHByaWNlKSAlPiUgDQogICAgICAgICAgICAgICAgICAgIGxheW91dCh0aXRsZSA9ICJLaW5nIENvdW50eSBIb3VzaW5nIEZlYXR1cmUgRGlzdHJpYnV0aW9ucyIpDQpwMiA8LSBwbG90X2x5KCkgJT4lIGFkZF9tYXJrZXJzKG5hbWUgPSAiUmF0aW5nIHYgUHJpY2luZyIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ID0gaG91c2luZ190cmFpbl9kYXRhJGdyYWRlLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGhvdXNpbmdfdHJhaW5fZGF0YSRwcmljZSkNCnAzIDwtIHBsb3RfbHkoKSAlPiUgYWRkX21hcmtlcnMobmFtZSA9ICJDb25kaXRpb24gdiBQcmljaW5nIiwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHggPSBob3VzaW5nX3RyYWluX2RhdGEkY29uZGl0aW9uLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGhvdXNpbmdfdHJhaW5fZGF0YSRwcmljZSkNCnA0IDwtIHBsb3RfbHkoKSAlPiUgYWRkX21hcmtlcnMobmFtZSA9ICJTcS4gRm9vdGFnZSB2IFByaWNpbmciLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeCA9IGhvdXNpbmdfdHJhaW5fZGF0YSRzcWZ0X2xvdCwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHkgPSBob3VzaW5nX3RyYWluX2RhdGEkcHJpY2UpDQoNCnN1YnBsb3QocDEsIHAyLCBwMywgcDQsIG5yb3dzID0gMikNCmBgYA0KTG9va2luZyBhdCB0aGUgZGlzdHJpYnV0aW9ucyBvZiBwcmljZXMgb3ZlciB0aGVpciBvcmRlcmluZyBieSBtYWduaXR1ZGUsIGl0IGFwcGVhcnMgdGhhdCB0aGUgcHJpY2VzIG9mIGhvdXNlcyB0aGF0IGFyZSBiZWxvdyBcJDEtMiBtaWxsaW9uIGZvbGxvdyBhIGxpbmVhciBkZWNsaW5lLCB3aXRoIGFsbCBob3VzZXMgYWJvdmUgXCQxLTIgbWlsbGlvbiBpbmNyZWFzaW5nIHNoYXJwbHkgaW4gdGhlaXIgcHJpY2VzIGFuZCBidWNraW5nIHRoZSB0cmVuZC4gIEFzc3VtaW5nIHRoYXQgdGhlIGZlYXR1cmVzIGluIHRoaXMgZGF0YXNldCBjYXB0dXJlIG1vc3Qgb2YgdGhlIGZhY3RvcnMgdGhhdCBpbmZsdWVuY2UgdGhlIHByaWNpbmcgb2YgYSBob21lLCBzb21lIGxpbmVhciBjb21iaW5hdGlvbiBvZiB0aGUgZmVhdHVyZXMgY291bGQgZ2l2ZSBhIHJlYXNvbmFibGUgZXN0aW1hdGUgZm9yIHRoZSBwcmljaW5nIG9mIGFuIGluZGl2aWR1YWwgaG91c2UgYmVsb3cgXCQxLTIgbWlsbGlvbi4gIFNpbmNlIHRoZSBkaXN0cmlidXRpb24gaXMgY2xvc2VyIHRvIGEgcG93ZXIgbGF3LCB0d28gc2VwZXJhdGUgbGluZWFyIGZ1bmN0aW9ucyBzaG91bGQgZ2l2ZSBhIHByZXR0eSBnb29kIGVzdGltYXRlIGJ5IGFjY291bnRpbmcgZm9yIHRoZSAna25lZScgaW4gdGhlIHByaWNpbmcgZGlzdHJpYnV0aW9uLiAgRmlyc3QgdGhvdWdoLCB3ZSdsbCB0cnkgZml0dGluZyBhIHNpbmdsZSBsaW5lIHRvIHRoZSBkYXRhIGFuZCBzZWUgd2hhdCBraW5kIG9mIGVycm9yIGNvbWVzIG91dC4NCg0KYGBge3IgbWVzc2FnZSA9IEYsIHdhcm5pbmcgPSBGfQ0KI1NjYWxlIGRhdGFzZXRzIA0KaG91c2luZ190cmFpbl9kYXRhX3NjYWxlZCA8LSBjYmluZChob3VzaW5nX3RyYWluX2RhdGFbLDFdLCBzY2FsZShob3VzaW5nX3RyYWluX2RhdGFbLDI6N10pKQ0KaG91c2luZ190ZXN0X2RhdGFfc2NhbGVkIDwtIGNiaW5kKGhvdXNpbmdfdGVzdF9kYXRhWywxXSwgc2NhbGUoaG91c2luZ190ZXN0X2RhdGFbLDI6N10pKQ0KDQpzdW1tYXJ5KGhvdXNpbmdfdHJhaW5fZGF0YV9zY2FsZWQpDQpzdW1tYXJ5KGhvdXNpbmdfdGVzdF9kYXRhX3NjYWxlZCkNCmBgYA0KDQpgYGB7ciBtZXNzYWdlID0gRiwgd2FybmluZyA9IEZ9DQpyZXF1aXJlKHRlbnNvcmZsb3cpDQoNCiNzcGxpdCBkYXRhIGZvciB0cmFpbmluZw0KI3Rlc3QgZ2VuZXJhbCBsaW5lYXIgbW9kZWwgb24gcmVsZXZhbnQgcXVhbnRpdGF0aXZlIGRhdGENCg0KI1Rlc3Qgb24gYSBudWVyYWwgbmV0d29yayBvZiAxMCBub2Rlcw0KDQoNClggPC0gdGYkcGxhY2Vob2xkZXIodGYkZmxvYXQzMiwgc2hhcGUoTlVMTCwgN0wpKQ0KWSA8LSB0ZiRwbGFjZWhvbGRlcih0ZiRmbG9hdDMyLCBzaGFwZShOVUxMLCAxTCkpDQoNCg0KDQpXMSA8LSB0ZiRWYXJpYWJsZSh0ZiR6ZXJvcyhzaGFwZSgxMCwgNykpKQ0KYjEgPC0gdGYkVmFyaWFibGUodGYkemVyb3Moc2hhcGUoMTAsIDEpKSkNClcyIDwtIHRmJFZhcmlhYmxlKHRmJHplcm9zKHNoYXBlKDEsIDEwKSkpDQpiMiA8LSB0ZiRWYXJpYWJsZSh0ZiR6ZXJvcyhzaGFwZSgxLCAxKSkpDQoNCkExIDwtIHRmJG5uJHJlbHUodGYkYWRkKHRmJG1hdG11bChYLFcxKSwgYjEpKQ0KQTIgPC0gdGYkbm4kcmVsdSh0ZiRhZGQodGYkbWF0bXVsKEExLFcyKSwgYjIpKQ0KDQpjb3N0IDwtIHRmDQpgYGANCg0KDQpgYGB7ciBtZXNzYWdlID0gRiwgd2FybmluZyA9IEZ9DQp0ZiRwbGFjZWhvbGRlcih0ZiRmbG9hdDMyLCBzaGFwZSg3TCwgTlVMTCkpDQoNCmhvdXNpbmdfdHJhaW5fcHJpY2UgPC0gIGhvdXNpbmdfdHJhaW5fZGF0YVssICJwcmljZSJdDQpob3VzaW5nX3Rlc3RfcHJpY2UgPC0gaG91c2luZ190ZXN0X2RhdGFbLCAicHJpY2UiXQ0KDQoNCmdsbV9tb2QgPC0gcnhGYXN0TGluZWFyKHByaWNlIH4gc3FmdF9saXZpbmcgKyBncmFkZSArIGNvbmRpdGlvbiwgZGF0YSA9IGhvdXNpbmdfdHJhaW5fZGF0YV9zY2FsZWQsIHR5cGUgPSAicmVncmVzc2lvbiIsIHZlcmJvc2UgPSAwKQ0KDQoNCnByaWNlX21vZCA8LSByeFByZWRpY3QoZ2xtX21vZCwgZGF0YSA9IGhvdXNpbmdfdGVzdF9kYXRhX3NjYWxlZCwgdmVyYm9zZSA9IDAsIGV4dHJhVmFyc1RvV3JpdGUgPSBjKCJzcWZ0X2xpdmluZyIsICJncmFkZSIsImNvbmRpdGlvbiIpKQ0KDQogeHZhbHMgPC0gMTpsZW5ndGgocHJpY2VfbW9kJFNjb3JlKQ0KDQpwcmljZV9lc3RfcGxvdGx5IDwtIHBsb3RfbHkoKSAlPiUgDQogIGFkZF9tYXJrZXJzKG5hbWUgPSAiVGVzdCBTZXQgUHJpY2UiLCB4ID0geHZhbHMsIHkgPSBob3VzaW5nX3Rlc3RfcHJpY2UkcHJpY2UpICU+JQ0KICBhZGRfbWFya2VycyhuYW1lID0gIkxpbmVhciBNb2RlbCBlc3QiLCB4ID0geHZhbHMsIHkgPSBwcmljZV9tb2QkU2NvcmUpICU+JQ0KICBhZGRfbGluZXMobmFtZSA9ICJNYWduaXR1ZGUgRGlmZmVyZW5jZSIsIHggPSB4dmFscywgeSA9IHByaWNlX21vZCRTY29yZSAtIGhvdXNpbmdfdGVzdF9wcmljZSRwcmljZSkgJT4lDQogIGxheW91dCh0aXRsZSA9ICJBdHRlbXB0aW5nIExpbmVhciBSZWdyZXNzaW9uIE1vZGVsDQogICAgICAgICAgICAgICAgICA8L2JyPiBmb3IgUHJpY2luZyBIb21lcyBpbiBLaW5nIENvdW50eSIpDQoNCiNyeExpbmVQbG90KFNjb3JlIH4gc3FmdF9saXZpbmcgKyBncmFkZSArIGNvbmRpdGlvbiwgdHlwZSA9IGMoInAiLCAiciIpLCBkYXRhID0gcHJpY2VfbW9kKQ0KcHJpY2VfZXN0X3Bsb3RseQ0KI3N1bW1hcnkoZ2xtX21vZCkNCmBgYA0KR2l2ZW4gdGhhdCB0aGUgZGlzdHJpYnV0aW9uIG9mIHByaWNlcyB3YXMgYWxyZWFkeSBrbm93biB0byBmb2xsb3cgYSBub24tbGluZWFyIGRpc3RyaWJ1dGlvbiwgdGhlIHJlc3VsdCBvZiB0aGlzIGxpbmVhciBlc3RpbWF0ZSBpc24ndCB2ZXJ5IHN1cnByaXNpbmcuICBUaGUgbGluZWFyIGZpdCBtYXkgaGF2ZSBiZWVuIHdpdGhpbiBhIGZldyBodW5kcmVkIHRob3VzYW5kIGRvbGxhcnMgZm9yIHRoZSBsb25nIHRhaWwsIGJ1dCB0aGUgd2hvbGUgc2VlbXMgdG8gYmUgY29ycnVwdGVkIGluIHRyeWluZyB0byBmaXQgdGhlIGNvbGxlY3Rpb24gb2YgdmFsdWVzIGFib3ZlIHRoZSBlbGJvdy4gIEJlbG93IGFyb3VuZCBcJDEgbWlsbGlvbiBkb2xsYXJzLCB0aGUgaG91c2luZyBwcmljZXMgYXJlIHByZXR0eSBldmVubHkgZGlzdHJpYnV0ZWQsIGJ1dCBhcyB0aGUgcHJpY2VzIGluY3JlYXNlIHRoZXkgc3RhcnQganVtcGluZyB1cCBieSBvcmRlcnMgb2YgbWFnbml0dWRlLiAgTGV0cyB0cnkgdGhlIHNhbWUgYXBwcm9hY2ggZm9yIGFsbCB2YWx1ZXMgb2YgcHJpY2UgPCAkNjAwSw0KDQpgYGB7cn0NCmhvdXNpbmdfdHJhaW5fZGF0YV9yZXYxIDwtIGZpbHRlcihob3VzaW5nX3RyYWluX2RhdGFfc2NhbGVkLCBwcmljZSA8IDhlNSkNCmhvdXNpbmdfdHJhaW5fcHJpY2VfcmV2MSA8LSBmaWx0ZXIoaG91c2luZ190cmFpbl9wcmljZSwgcHJpY2UgPCA4ZTUpDQpob3VzaW5nX3Rlc3RfZGF0YV9yZXYxIDwtIGZpbHRlcihob3VzaW5nX3Rlc3RfZGF0YV9zY2FsZWQsIHByaWNlIDwgOGU1KQ0KaG91c2luZ190ZXN0X3ByaWNlX3JldjEgPC0gZmlsdGVyKGhvdXNpbmdfdGVzdF9wcmljZSwgcHJpY2UgPCA4ZTUpDQptYXhfdGVzdF9yZXYxIDwtIGFwcGx5KGhvdXNpbmdfdGVzdF9kYXRhX3JldjEsIEZVTiA9IG1heCwgTUFSR0lOID0gMikNCm1pbl90ZXN0X3JldjEgPC0gYXBwbHkoaG91c2luZ190ZXN0X2RhdGFfcmV2MSwgRlVOID0gbWluLCBNQVJHSU4gPSAyKQ0KbWVhbl90ZXN0X3JldjEgPC0gYXBwbHkoaG91c2luZ190ZXN0X2RhdGFfcmV2MSwgRlVOID0gbWVhbiwgTUFSR0lOID0gMikNCg0KDQpnbG1fbW9kX3JldjEgPC0gcnhGYXN0TGluZWFyKHByaWNlIH4gc3FmdF9saXZpbmcgKyBncmFkZSArIGNvbmRpdGlvbiwgZGF0YSA9IGhvdXNpbmdfdHJhaW5fZGF0YV9yZXYxLCB0eXBlID0gInJlZ3Jlc3Npb24iLCB2ZXJib3NlID0gMCkNCg0KDQpwcmljZV9tb2RfcmV2MSA8LSByeFByZWRpY3QoZ2xtX21vZF9yZXYxLCBkYXRhID0gaG91c2luZ190ZXN0X2RhdGFfcmV2MSwgdmVyYm9zZSA9IDAsIGV4dHJhVmFyc1RvV3JpdGUgPSBjKCJzcWZ0X2xpdmluZyIsICJncmFkZSIsImNvbmRpdGlvbiIpKQ0KeHZhbHNfcmV2MSA8LSAxOmxlbmd0aChwcmljZV9tb2RfcmV2MSRTY29yZSkNCg0KcHJpY2VfZXN0X3Bsb3RseV9yZXYxIDwtIHBsb3RfbHkoKSAlPiUgDQogIGFkZF9tYXJrZXJzKG5hbWUgPSAiVGVzdCBTZXQgUHJpY2UiLCB4ID0geHZhbHNfcmV2MSwgeSA9IGhvdXNpbmdfdGVzdF9wcmljZV9yZXYxJHByaWNlKSAlPiUNCiAgYWRkX21hcmtlcnMobmFtZSA9ICJMaW5lYXIgTW9kZWwgZXN0IiwgeCA9IHh2YWxzX3JldjEsIHkgPSBwcmljZV9tb2RfcmV2MSRTY29yZSkgJT4lDQogIGFkZF9saW5lcyhuYW1lID0gIk1hZ25pdHVkZSBEaWZmZXJlbmNlIiwgeCA9IHh2YWxzX3JldjEsIHkgPSBwcmljZV9tb2RfcmV2MSRTY29yZSAtIGhvdXNpbmdfdGVzdF9wcmljZV9yZXYxJHByaWNlKSAlPiUNCiAgbGF5b3V0KHRpdGxlID0gIkF0dGVtcHRpbmcgTGluZWFyIFJlZ3Jlc3Npb24gTW9kZWwNCiAgICAgICAgICAgICAgICAgIDwvYnI+Zm9yIFByaWNpbmcgSG9tZXMgaW4gS2luZyBDb3VudHkNCiAgICAgICAgICAgICAgICAgIDwvYnI+V2hlcmUgUHJpY2Ugb2YgSG9tZSA8ICQ4MDBrIikNCg0KcHJpY2VfZXN0X3Bsb3RseV9yZXYxDQojc3VtbWFyeShnbG1fbW9kX3JldjEpDQpgYGANCkJ5IEN1dHRpbmcgb2ZmIHRoZSBsZXNzIGxpbmVhciBwYXJ0IG9mIHRoZSBkYXRhc2V0LCB0aGUgbWF4ZXJyb3IgYmV0d2VlbiB0aGUgZXN0aW1hdGUgYW5kIHRoZSBhY3R1YWwgcHJpY2Ugd2FzIGRlY3JlYXNlZC4NCmBgYHtyIG1lc3NhZ2UgPSBGLCB3YXJuaW5nID0gRn0NCg0KDQoNCmBgYA0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K